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Abstract 

This paper discusses a planner-based approach to 
automating data production tasks, such as pro- 
ducing fire forecasts from satellite imagery and 
weather station data. Since the set of available data 
products is large, dynamic and mostly unknown, 
planning techniques developed for closed worlds 
are unsuitable. We discuss a number of techniques 
we have developed to cope with data production do- 
mains, including a novel constraint propagation al- 
gorithm based on planning graphs and a constraint- 
based approach to interleaved planning, sensing 
and execution. 

1 Introduction 

Petabytes of remote sensing data are now available from 
Earth-observing satellites to help measure, understand and 
forecast changes in the Earth system, but using these data ef- 
fectively can be surprisingly hard. The volume and variety of 
data files and formats are daunting. Simple data management 
activities, such as locating and transferring files, changing 
file formats, gridding point data, and scaling and reproject- 
ing gridded data, can consume far more personnel time and 
resources than the actual data analysis. We address this prob- 
lem by developing a planner-based agent for data production, 
called IMAGEbot [Golden et al . , 2003], that takes data prod- 
uct requests as high-level goals and executes the commands 
needed to produce the requested data products. 

The data production problem consists of converting an ini- 
tial set of low-level data products into higher-level data prod- 
ucts that can be used for science or decision support. The data 
products we are concerned with are geospatial data measuring 
specific variables of the Earth system, such as precipitation, 
vegetation productivity and fire risk, but our approach is also 
applicable to other types of data. Higher-level data products 
may be transformed versions of lower-level data products, 
or they may be entirely new products providing estimates or 
predictions of unknown Earth system variables, such as soil 
moisture, based on known variables, such as precipitation. 
These variables are estimated by running one or more com- 
putational models , such as simulation codes. The models can 
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be precisely characterized in terms of their input and output 
requirements, which makes them straightforward to represent 
in an AI planning system. However, there are significant dif- 
ferences between the data production problem and more tra- 
ditional planning domains, calling for different techniques. 

Notable features of data processing domains include large 
dynamic universes, incomplete information and uncertainty. 
TTiere are petabytes of data available, with new data becom- 
ing available all the time, and the agent itself produces many 
new data products in the course of fulfilling the user’s goal — 
data products that could be used to fulfill subsequent goals. 
There is also considerable uncertainty — uncertainty of the 
time that particular data will be available, or whether the data 
will arrive at all, uncertainty in the quality of data, even un- 
certainty as to whether a given processing algorithm will suc- 
ceed. To cope with this uncertainty, the agent may need to 
poll for data availability or try alternative courses of action if 
the one it is pursuing seems unpromising. 

We have developed a planner-based agent, called IMAGE- 
bot, to automate data production. The data production prob- 
lem may be viewed as a planning problem in which the initial 
state describes the current set of available data products, and 
whose goal state describes the properties of the desired high- 
level data products. Planner operators correspond to data 
transformation and generation tools. IMAGEbot takes data 
product requests as high-level goals and executes the com- 
mands needed to produce the requested data products. 

We adopt a planning approach somewhat similar to Graph- 
plan, consisting of a Graphplan-style reachability analysis 
and a constraint-based search. However, the large universe 
of the data production problem makes the grounded planning 
graph of Graphplan inapplicable; instead, we choose a lifted 
representation where actions and plans contain variables. Be- 
cause of the lifted representation, and the uncertain and dy- 
namic nature of the data production problem, the reachability 
analysis and search cannot be separated; instead, IMAGEbot 
is interleaves planning, constraint reasoning and execution. 

In this paper, we report on our work on IMAGEbot, with 
a focus on the constraint reasoning that underlies planning, 
sensing and execution. Section 2 gives an overview of the 
IMAGEbot system architecture and high-level planning ap- 
proach; Section 3 discusses our constraint-based approach to 
sensing; Section 4 discusses a novel constraint propagation 
algorithm based on the planning graph. Section 5 discusses 



Figure 1 : The architecture of IMAGEbot 


interleaved planning and execution. 

2 IMAGEbot Overview 

2.1 System Architecture 

The architecture of the IMAGEbot agent is depicted in Figure 
1 . The main components are: 

JDAF: The Java Distributed Application Framework com- 
prises execution environment for IMAGEbot; it provides the 
agent with a common API for data-processing programs and 
ecological forecasting models. 

DPADL: The Data Processing Action Description 
Language [Golden, 2002] is used to provide action descrip- 
tions of data-processing programs and available data sources. 
Goals, in the forms of data product requests, can also be de- 
scribed in DPADL. To support both fine-grained and flexible 
sensing, DPADL allows constraints to make calls calls to the 
underlying runtime environment (Section 3). 

DoPPLER: The Data Processing Planner accepts goals in 
the form of data descriptions and synthesizes and executes 
data-flow programs. It reduces the planning problem to a CSP 
whose solution provides a solution to the original planning 
problem. 

JNET: Java Constraint Network is a constraint represen- 
tation and reasoning framework that provides the agent with 
constraint propagation and search capabilities. 

The architecture provides a planning framework that inter- 
leaves planning with constraint reasoning and plan execution. 

2.2 Planning Approach 

Planning in IMAGEbot is a two-stage process. The first stage 
consists of a Graphplan-style reachability analysis [Blum & 
Furst, 1997] to derive heuristic distance estimates for the sec- 
ond stage, a constraint-based search. These stages are not en- 
tirely separate, however; constraint propagation occurs even 
in the the graph-construction stage, and the graph is refined 
during the constraint-search phase. 

Lifted planning graphs 

Planning domains are specified in DPADL. From the plan- 
ning problem specification, the planner incrementally con- 
structs a directed graph, similar to a planning graph [Blum 
& Furst, 1997], but using a lifted representation contain- 
ing variables). This graph is used to obtain distance estimates 
for heuristic search, and is also the basis for the construction 
of the CSP Arcs in the graph are analogous to causal links 
[Penberthy & Weld, 1992]. A causal link is triple (a s ,p, a p ), 


recording the decision to use action a s to support precondi- 
tion p of action a p . However, instead of recording a commit- 
ment of support, it indicates the possibility that a s supports p. 
The lifted graph contains multiple ways of supporting p; the 
choice of the actual supporter becomes a constraint satisfac- 
tion problem. We add an extra term to the arc for bookkeep- 
ing purposes - the condition, 7 p s , needed in order for a s to 
achieve p. A link then becomes (a s , yg 3 ,p, a p ). 

Given an unsupported precondition p of action a p , our first 
task is to identify all the actions that could support p. Be- 
cause the universe is large and dynamic, identifying all possi- 
ble ground actions that could support p would be impractical, 
so instead we use a lifted representation, identifying all action 
schemas that could provide support. Given an action schema 
a, we determine whether it supports p by regressing p through 
a. The result of regression is the formula y p s . If 7 p 3 =-L, 
then a does not support p. Initial graph construction termi- 
nates when all preconditions have support or (more likely) a 
potential loop is detected. 

From planning to constraints 

After the graph is constructed, heuristic distance estimates 
for guiding the search are computed, and a constraint prob- 
lem representing the search space is incrementally built. It 
is incremental because the planning graph comprises a com- 
pact representation of the search space, in which each action 
node can represent multiple concrete actions in the final plan. 
Since the number of possible actions can be large, even infi- 
nite, we cannot simply generate all of them at once but do so 
lazily during search. This is handled using a dynamic CSP 
(DCSP), in which new variables and constraints can be added 
for each new action and causal link in the plan. 

The CSP contains: 1 ) boolean variables for all arcs, nodes 
and conditions; 2 ) variables for all parameters, input and out- 
put variables and function values; 3) for every condition in 
the graph, a constraint specifying when that condition holds 
(for conditions supported by arcs, this is just the XOR of 
the arc variables); 4) for conjunctive and disjunctive expres- 
sions, the constraint is the respective conjunction or disjunc- 
tion of the boolean variables corresponding to appropriate 
sub-expressions; 5) for every arc in the graph, constraints 
specifying the conditions under which the supported fluents 
will be achieved (i.e., 7 p => p, where 7 ^ is the precondition 
of a needed to achieve p) ; 6 ) user-specified constraints; and 
7) constraints representing structured objects. 

Constraint-based search 

After converting the planning problem to a CSP, the planner 
searches the CSP for a solution. At a high level, the plan- 
ner, guided by heuristic distance estimates extracted from the 
planning graph, selects subgoals to achieve and actions to 
achieve them (Algorithm 2). After the subgoal and action 
selection, the planner (or more accurately, the CSP solver) 
finds values for variables representing planner action param- 
eters. This is necessary to make actions executable. During 
the search, propagation is performed whenever a value is as- 
signed to a variable. The search is an iterative process involv- 
ing possible backtracks; that is, if there are no valid parame- 
ters for a chosen action, the planner has to search for another 
plan; if it is impossible to extract a plan from the current plan 









graph, the planning graph is extended or search fails. 

3 Constraint-based sensing 

In order to find out what data products relevant to the task at 
hand are available, the agent needs to sense its environment. 
One way of doing this is to introduce sensing actions [Golden 
& Weld, 1996], which the agent can execute in order to obtain 
information. This approach has the advantage that it can be 
used to capture sensing actions that have preconditions, but it 
also requires the plan to be at least partially executed before 
the information can be obtained. We follow an alternative ap- 
proach of representing low-cost precondition-free sensors us- 
ing procedural constraints . That is, we can implement con- 
straints as procedures that can perform database queries or 
invoke other information-gathering operations in the course 
of identifying the domain of values a given variable can have. 
This constraint-based sensing approach is much more flexible 
than the sensing-action approach, as the order of sensing op- 
erations is based on constraint propagation, and information 
dependencies are inherently multi-directional. For example, 
suppose we have a set of satellite images, each of which cor- 
responds to a given region of the Earth’s surface for a given 
day. If we have a specific satellite image, we may invoke 
methods to determine the region and day for that image. On 
the other hand, if we know we need an image corresponding 
to a given region and day, we may query a database to find 
out what images are available for the time and place in ques- 
tion. The specific set of operations performed depends on 
which variables are bound (or appropriately restricted), i.e., 
what information is “known” to the constraint solver. Invok- 
ing a sensing operation may trigger further sensing through 
constraint propagation. For example, suppose we are inter- 
ested finding a high resolution satellite image of western Ore- 
gon for a day in June that had no rainfall. We can perform 
a database query to find out what images are available over 
western Oregon for June. Once the images are known, we can 
query to find the resolution of each one, eliminating from con- 
sideration those of insufficient resolution. We can then query 
to determine the day that each image was captured, then do 
another query to determine the precipitation for that day. Fi- 
nally, we remove from consideration all images for days that 
had non-zero precipitation. The order of sensing operations 
depends on what information is “known” and what informa- 
tion is needed. 

We can also represent more traditional sensing actions, us- 
ing actions that produce new objects (data files), which con- 
tain information. Acquiring these objects can, in turn, trig- 
ger more constraint propagation, resulting in more implicit 
sensing. For example, a data-acquisition action may obtain 
a set of satellite images from a remote location. Once these 
images are available, additional operations can be performed 
to obtain information about the images, such as data qual- 
ity. These additional operations can be implemented as con- 
straints rather than actions, which removes them from the set 
of deliberate decisions that the planner needs to make. 

4 Action-based Constraint Propagation 

As we have discussed, data production problems, due to their 
large, uncertain and dynamic universes, are not suitable for 


a grounded representation. The lifted planning graph is a 
much more concise representation than the grounded plan- 
ning graph, but it is potentially less informative, which makes 
conventional constraint propagation and search less effective. 
The CSP derived from the lifted planning graph contains vari- 
ables with infinite domains [Golden & Frank, 2002], so there 
is no way to enumerate solutions by search alone, yet the tra- 
ditional constraint propagation that establishes certain levels 
of consistency does not work well either. For example, we 
have a constraint propagator in JNET that enforces a partial 1 
generalized arc -consistency (GAC) [Bessiere & Ch, 1997; 
Katsirelos & Bacchus, 2001]. The definition of GAC is built 
upon the variables and their values; namely, a CSP is GAC if 
all its variables are GAC; a variable is GAC is all its values 
are GAC; a value v of a variable x is GAC if it has support 
from other variables in every constraint on x. Establishing 
consistency requires evaluating every value to see if it satis- 
fies certain constraints, which is not possible in general for 
infinite variable domains. A combination of propagation and 
search will eventually find a solution, but propagation does 
not become informative until late in the search . 

We have developed a new constraint propagation algorithm 
that propagates changes among the actions in the planning 
graph, which yields much more information, even before 
search begins. It not only restricts the domains of variables 
by eliminating inconsistent values, but it also may add values 
to the variable domains when new information is available 
(e.g., a new object is created). In this section, we first de- 
scribe the propagation algorithm, then illustrate how it works 
with an example, and discuss its role in the planning search 
and constraint search. 

4.1 Algorithm 

Formally, a data-processing action schema can be seen as a 
tuple (X, O, V, n, £, x)> where 1,0, V are the input variables, 
output variables and parameters respectively. The parame- 
ters are unknowns that may appear in constraints on either or 
both input and output. II is the precondition , £ is effects and 
X is a procedure for executing the action that may reference 
any variable in X U V and must set every variable in O. A 
lifted planning graph can be seen as a partially ordered set 
of actions (A, -<), where a ^ b iff action a supports b or a 
supports c and c -< b. In the CSP derived from the lifted plan- 
ning graph, we have constraints specifying the relationships 
among variables inside an action and constraints specifying 
relationships of two actions if one supports another. For an in- 
dividual action, if something changes, for example, if a value 
is assigned to a variable in the action input due to search , the 
change to this variable can be propagated to other variables in 
the output, which may change their domains. For two actions 
a and 6, where a supports 6, changes in the input of b can be 
propagated to the output of a; similarly, changes in the out- 
put of a can be propagated to the input of b. The idea of this 
propagation is outlined in Algorithm 1. 

In Algorithm 1, function enforce(P, C a ) enforces ev- 
ery constraint c € C a associated with action a. It restricts 

1 We call it partial GAC for two reasons: 1) not every constraint 
procedure enforces the GAC; and 2) not every constraint is executed 
in the propagation. 


Algorithm 1 Action Constraint Propagation 
Given a lifted plan graph G. Let A be the set of actions in G, 
let P = (X, D, C) be the CSP derived from the lifted plan 
graph, and let A! be a subset of actions to be propagated: 

propagate(G, A, P, A') 

1 . while ( A 1 ^ 0) do 

(a) let a <— an action removed from A' 

(b) let C a <— constraints relevant to a 

(c) < d(l(a)),d(0(a)) ><— enforce(P,C a ) 

(d) for (Vz E 1(a) s.t. d(i) = 0) 
remove supporting link to z 

(e) for (Vo E 0(a) s.t. d(o) = 0) 
remove supporting link from o 

(f) for (Vz E 1(a) s.t. d(i) changed) 
i. for (V6 E A s.t. b supports a) 

if (revise(P,G(6),z)) A! <- A'U {6} 

(g) for (Vo E 0(a) s.t. d(o) changed) 
i. for (V6 E A s.t. a supports b) 

if(revise(P,l(b),o))A' <- A'U {b} 

2. return 


domains of variables in c by eliminating inconsistent values. 
Function revise(P, 0(b), z) (or revise(P, X(6), o) ) com- 
putes the domains of variables in G(b) (or 1(b) ), where z is 
an input (or o an output) of action a and action b supports (or 
is supported by) a. The function revise may remove in- 
consistent values or add newly discovered values depending 
on the planning graph structure. It returns true if any variable 
domain has been revised, in which case the action b is added 
to A\ waiting to be propagated. 

In addition to removing inconsistent values or discovering 
new values for variables in an action, this propagation also 
removes certain supporting links if it identifies inconsistency. 
If all links from an action a supporting other actions are re- 
moved, the action a is useless in the planning graph so it can 
be safely removed. If all links to an input of an action a 
are removed, this action cannot be executed because one of 
its inputs does not have support. The planner either has to 
find other support for this action (e.g., expanding the plan- 
ning graph by inserting more actions) or remove this action 
from the planning graph. 

4.2 Example 

For illustration, we consider a simplified version of construct- 
ing a mosaic. Many satellites continuously image whatever 
portion of the Earth they pass over, like giant hand-held scan- 
ners. For convenience, the resulting swath data is usually re- 
projected into onto a 2D map and chopped up into tiles , cor- 
responding to a regular grid drawn over the map. To obtain 
the data pertaining to a particular region of the Earth, we first 
identify and obtain the tiles that cover that region and then 
combine them into a single image, known as a mosaic, and 
crop away the pixels outside the region of interest. 

These tiles are represented in the planner as first-class ob- 
jects. The attributes of a tile describe, among other things. 




Figure 2: The planner actions: the dots inside actions are 
inputs and outputs. Parameters are not shown. 


the physical measurement the data in the tile represent, the 
position of the tile on the grid, the projection used to flatten 
the globe, and the region of the Earth covered by the pixels 
in the image. For simplicity, we assume in this example that 
tiles have only two attributes: the region a tile covers and 
the cloudiness when the image was taken. A simplified task 
becomes to take some tiles from thousands of available tiles 
and compose them to create a mosaic that covers a specified 
region without too much cloud cover. 

Specifically, a region is a pair of points (ul, Ir) where ul 
is the upper-left comer and Ir the lower-right comer. A point 
is a pair of coordinates (x,y). Normally x and y would be 
longitude and latitude, but as a further simplification, we will 
assume both x and y are non-negative integers. The cloudi- 
ness is represented by a real number from 0 to 1, where 0 
is clear sky and 1 is totally obscured. Further, we assume 
there are only three actions the planner may take: compose 
two tiles horizontally ( comp2h ) or vertically ( comp2v ), or get 
a tile with its ul point as a parameter (getTile). A real mo- 
saic command is not limited to combining two tiles. Figure 2 
shows action preconditions and effects with respect to the re- 
gion. In addition, the effect of composing two images is that 
their combined cloudiness is treated as the maximum of the 
cloudiness of the input tiles. 

A problem instance we consider here consists of some 
small tiles, such as ((0, 0), (1, 2)), or ((2, 3), (3, 5)). The goal 
is to compose a mosaic for the region ((0, 0), (3, 2)) with no 
more than 15% cloud cover. This mosaic is composed of tiles 
Bi t Pi , ..., Be , which may or may not available locally; if 
not, we assume that action getTile((x , y )) can be executed to 
get any available tiles ((x, z/), (x 4- ra, y 4- n)). 

The planning graph created by the planner is shown in Fig- 
ure 3, where nodes represent lifted actions and arcs the sup- 
porting relations. The dots inside action nodes are inputs and 
outputs of the actions, each representing a set of objects, pos- 
sibly infinite. At the time when a CSP is derived from this 
planning graph, these unknown objects, inputs and outputs of 
the actions and their parameters, are represented as variables 
with infinite domains. 

The action-based constraint propagation can be invoked 
to restrict some of the infinite domains. Since the plan- 
ner goal ((0,0), (3,2)) is known, the output of action 
comp2v, which supports the goal, is also known; apply- 
ing the propagation on cora/?2v, we have the domains 





Figure 3: A planning graph 

of its two inputs, both of which are singletons, namely 
{((0,0), (3, 1))} and {((0, 1), (3,2))}. Similarly, the output 
of comp2h is known; applying the propagation on comp2h, 
we have the domains of its two inputs, both of which 
contain two regions: {(( 0 , 0 ), ( 1 , 2 )) , (( 0 , 0 ), ( 2 , 2 ))} and 
, {((1, 0), (3, 2)) , ((2, 0), (3, 2))}, respectively. The changes 
to inputs of the these actions are propagated to the next level 
actions supporting them. 

When propagation stops, we have a much more limited 
search space as shown in Figure 4, where the tiles in the in- 
puts and outputs are restricted to specified regions. These tiles 
crossed out are the ones eliminated by propagation from the 
initial state. Notice also that many links appearing in Figure 
3 have been removed by the propagation. For example, all 
links from comp2v to comp2v have been removed. 

We also have a goal constraint requiring the cloudiness 
of the image to be at most 15%. Propagating this* con- 
straint backward through the graph results in the requirement 
that each input image has a cloudiness of at most 15% (not 
shown). However, the cloudiness of the tiles is unknown at 
planning time, so no further propagation or pruning can be 
done until the plan is at least partially executed. We continue 
with this example in the next section. 

5 Planning and Execution 

Although our constraint-based approach to sensing helps to 
cope with large, unknown domains, there is still some uncer- 
tainty, even for a “complete” plan. Data products may turn 
out to be of a lesser quality than expected, due to cloud cover 
for instance, or may even turn out to be missing entirely. Pro- 
cessing algorithms may fail to perform as well as expected, 
perhaps due to problems with the input data, or they may 
simply crash. Some quality problems can be automatically 
detected, but only after the data products are in hand, meaning 
after the plan has been at least partially executed. Fortunately, 
the non-destructive nature of data production domains means 
the cost of plan execution is limited to the time and resources 
consumed, so it is natural to view plan execution as an ex- 
tension of the search process. If partial execution of a plan 
reveals a violation of a constraint or preference, it is a simple 
matter to backtrack and try something else, since there are 



Figure 4: Constraint propagation in the planning graph. Ob- 
jects in a dotted rectangles are inputs to an action; an object 
divided by dashed line is a composed object; single objects 
are available in the initial state or can be obtained with get- 
Tde (not shown). 


Algorithm 2 Plan construction and execution. Iteratively 
supports subgoals and executes actions until all goals are sup- 
ported and all actions are executed. The keyword pick indi- 
cates a choice that is not a backtrack point. The keyword 
choose indicates nondeterministic choice (backtrack point) 
The keyword fail indicates a backtrack, 
public void PlanAndExecute(goal, actions) 

1 . let G <— BuildPlanGraph(goal, actions) 

2. let P <— BuildConstraintNet(GO, A <— Actions in G 

3. let agenda <— {goal}, unexecuted <— {goal} 

4. set d(goal) <— {true} 

5. while (propagate(G, A, P, A) returns false) 

if (ExpandGraph(<2, P ) returns false) fail 

6 . while (unexecuted ^ 0 ) pick 

(a) pick a ^unexecuted 

if (execute(a) returns true) 

remove a from unexecuted 

(b) let p <— remove from agenda 

i. choose (a s , 7 ^ s ,p,a p ) in G 

ii. add 7 "* to agenda and set d{^ s ) = {true} 

iii. add a s to unexecuted 

iv. if (propagate(G, A, P, {a s ,a p }) returns false) 

fail 

(c) ExpandGraph(G, P) 



Figure 5: Partial execution provides additional information, 
which allows additional constraint propagation and pruning 
of the planning graph. 

no state changes to be undone. Furthermore, actions may be 
executed before the plan is complete, yielding information to 
reduce search or choose between competing options. For ex- 
ample, if there are two candidate data sets, each of unknown 
quality and each of which requires different processing steps, 
the planner can execute the actions to obtain both sets of data 
and decide which one to use before wasting time planning out 
all the processing operations for data that may not be used. 

Here, again, the planning graph representation is useful, 
because it provides a guide to which data sources and actions 
are relevant to a problem without requiring a complete plan to 
be generated. Once an action has been executed and its out- 
puts produced, the output variables are instantiated with the 
results from execution and the constraints are re-propagated, 
which may further restrict the domains of other variables, re- 
ducing the amount of search. 

5.1 Example 

To continue our previous example, suppose that we execute 
all the getTile actions in the planning graph before doing any 
explicit search. Since getTile obtains the actual images, con- 
straint propagation will result in determining the cloudiness 
of each of the images. Recall that the domain for each cloudi- 
ness variable was [0 . . . 0.15], since the maximum allowable 
cloudiness specified in the goal is 0.15,. During propagation, 
the actual cloudiness of each tile will be determined and in- 
tersected with the original domain of [0 . . . 0.15]. If the value 
is greater than 0. 15, the domain will become empty. Suppose 
the tile spanning B 1 and B2 has cloudiness of 0.25, and all 
the others have cloudiness of 0.0. This result is propagated 
through the action graph, eliminating a number of values and 
two actions (Figure 5). 

6 Conclusions 

IMAGEbot is implemented and has been integrated into 
an ecological forecasting application [Golden et al., 2003], 
which produces “nowcasts” and forecasts of socioeconomic 
importance, such as crop health and fire risk. 


We believe the constraint-based sensing and planning- 
graph propagation approaches introduced in this paper would 
be equally suitable to other software domains that involve 
large, unknown dynamic domains. Related applications to 
which planners have been applied include Internet softbots 
[Golden, 1998; Etzioni, Golden, & Weld, 1997], web ser- 
vices [Srivastava & Kholer, 2003], image processing [Lansky, 
1998; Chien et al, 1997], and grid-based computing [Blythe 
et al, 2003]. 
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